Materials and methods for simulating focal shifts in viewers using large depth of focus displays

ABSTRACT

A large depth of focus (DOF) display provides an image in which the apparent focus plane is adjusted to track an accommodation (focus) of a viewer&#39;s eye(s) to more effectively convey depth in the image. A device is employed to repeatedly determine accommodation as a viewer&#39;s gaze within the image changes. In response, an image that includes an apparent focus plane corresponding to the level of accommodation of the viewer is provided on the large DOF display. Objects that are not at the apparent focus plane are made to appear blurred. The images can be rendered in real-time, or can be pre-rendered and stored in an array. The dimensions of the array can each correspond to a different variable. The images can alternatively be provided by a computer controlled, adjustable focus video camera in real-time.

RELATED APPLICATIONS

This application is the U.S. national phase of internationalapplication, 371 of PCT/US03/07214, filed Mar. 7, 2003.

This application is based on a co-pending provisional application, U.S.Ser. No. 60/365,130, filed on Mar. 15, 2002, the benefit of the filingdate of which is hereby claimed under 35 U.S.C. § 119(e).

FIELD OF THE INVENTION

The present invention generally pertains to a method and a display forconveying depth in a real or virtual image, and more specifically,pertains to selectively applying blur cues to images portrayed on alarge depth of focus (DOF) display, based on the accommodation (focus)or vergence of the viewer's eye, to increase the perception of depth bythe viewer.

BACKGROUND OF THE INVENTION

The world, as we experience it, is three-dimensional (3-D). However,although we experience a 3-D world, our senses do not directly receive3-D data about the world. Instead, the optics of each eye project atwo-dimensional (2-D) image onto the surface of the retina, and thevisual system must infer data about the missing third dimension (i.e.,depth) from those 2-D images and from various supplementary depth cues.These depth cues include the oculomotor cues of vergence andaccommodation, and the stereoscopic cue of binocillar disparity.

Vergence: When a person's gaze is shifted to an object, the person'seyes move to fixate on the object, that is, to place the retinal imageof that object on the center of each eye's retina (the fovea), where theresolution of the eye is the highest. Oculomotor cues involve thesensing of the position of muscles in and around the eyes. Oneoculomotor cue, vergence, refers to the phenomenon that lines of sightof the eyes are approximately parallel to one another when they arefixating on a very distant object, and the eyes rotate in toward eachother (i.e., converge) as they fixate on closer objects. The brainreceives sensory feedback regarding the relative eye positions, and thisinformation serves as a depth cue.

Accommodation: Like most cameras, the human eye has a limited DOF. Whenviewing a real scene, not every object in the scene is in focus at anygiven time. Instead, the viewer accommodates (adjusts the focus of theeye) to bring objects at various distances into focus. For instance, ifthe viewer accommodates to an object that is one meter away, the retinalimage of an object that is 20 meters away is blurry. The farther away anobject is from the focus point of the viewer, the blurrier the retinalimage of that object is.

The eye possesses a two part optical system. The cornea provides themajority of refraction (approximately 70 percent), but its refractivepower is fixed. The crystalline lens sits behind the cornea, and itsshape can be altered to increase or decrease its refractive power.

When a fixated object is close to the observer, the ciliary muscles ofthe eye contract in order to make the crystalline lens more sphericaland increase its refractive power, so that the image of the object isbrought into focus on the retina. When a fixated object is far from theobserver, the ciliary muscles of the eye relax, thereby flattening thelens and decreasing its refractive power (measured in diopters) so thatthe image of the object is brought into focus on the retina. Dioptricblur provides negative feedback that the accommodation control systemuses when trying to accommodate to an object at a given distance at agiven point in time. If a person looks at an object at a novel depth, itwill be blurred from the initially inaccurate state of accommodation. Ifthe system begins to shift accommodation in one direction, and theobject becomes more blurry, this blur feedback causes the system toreverse the direction of the accommodation shift. If, instead, theobject becomes clearer, then accommodation continues to shift in thesame direction. If the shift in accommodation overshoots the point ofbest focus, this manifests as increased blur, and the shift inaccommodation reverses direction and slows. These shifts inaccommodation continue until the blur feedback is minimized (the objectcomes into best focus). The process is dynamic, and the eye constantlymonitors blur feedback and makes corrections in accommodation at ratesup to 5 Hz. This process of natural viewing and focusing is known asclosed-loop accommodation, because the blur feedback loop is intact (or“closed”). The brain receives feedback about the state of activity ofthe ciliary muscles, providing the viewer with information about thedepth of the object being viewed.

Some viewing conditions artificially increase the DOF of the eye. Forinstance, if a scene is viewed through a small pinhole, then bothdistant and near objects are in focus at the same time. Under suchconditions, the negative feedback of dioptric blur is removed orsubstantially decreased, and accommodation is said to be “open-loop”(because the feedback loop is interrupted). Under open-loopaccommodation conditions, the viewer can accommodate from extremely nearto far without a significant change in the retinal image of the scene.Some video displays can be made to have a very large DOF. As oneexample, the virtual retinal display (VRD) described in U.S. Pat. No.5,467,104 can have a large DOF, producing an open-loop accommodativeresponse in users. Other large DOF displays can be fabricated andmethods presented in this document are applicable to all possible largeDOF displays.

Vergence and Accommodation are Synkinetic: When one shifts one's gaze toan object at a given depth, the resultant vergence and accommodationresponses are highly correlated. Not surprisingly, the accommodation andvergence mechanisms are synkintetic (an involuntary movement in accordwith one mechanism is triggered when a movement in accord with the othermechanism occurs). This linkage can be observed in infants between threeto six months old, suggesting a biological predisposition for thesynkinesis. When the eye accommodates to a certain depth, the vergencesystem is automatically driven to converge to the same depth.Conversely, when the eye converges to a certain depth, the accommodationsystem is automatically driven to accommodate to the same depth. Thesecross couplings between accommodation and vergence are referred to asconvergence driven accommodation and accommodation driven vergence.

Binocular Disparity and Stereopsis: Another depth cue is binoculardisparity. Because a small distance separates the two eyes, they haveslightly different viewpoints, and hence different retinal images. Instereopsis, the visual system compares the images from the left andright eye, and makes inferences about the depth of objects based ondisparities between the retinal locations on which the images of theobjects fall. This depth cue has been exploited in stereographicdisplays (including Head Mounted Displays (HMDs)), which presentdifferent images to each eye.

An object at the point of fixation falls on corresponding points of theretina (the center of the fovea, in this case). Other objects atapproximately the same depth as the fixated object will also fall oncorresponding points of the retina. The imaginary curved plane thatdescribes the area of space that will fall on corresponding retinalpoints is referred to as the horopter. Objects behind the horopter willbe shifted toward the left side of the retina in the right eye andtoward the right side of the retina in the left eye (i.e., the imagesare shifted toward the nose). Objects in front of the horopter will beshifted toward the right side of the retina in the right eye and towardthe left side of the retina in the left eye (i.e., the images areshifted toward the ears).

Interaction between Accommodation, Vergence, and Stereopsis: All ofthese depth cues interact. As mentioned previously, accommodation andvergence are synkinetic. Vergence and stereopsis interact. In order tostereoscopically fuse objects at different distances, the eyes mustconverge to fixate upon those objects. The relative distance betweenright and left object images is greater for objects in the foreground ofa stereographic image than for objects in the background of the image.As viewers use stereographic displays and look at objects in theforeground and background of the displayed scene, they must dynamicallyshift vergence.

Accommodative Response to Current Non-Stereographic Video Displays:Research has indicated that viewers do not accurately focus their eyeson standard (non-stereographic) video displays (e.g., liquid crystaldisplays (LCDs) and cathode ray tubes (CRTs)). Their focus is, instead,biased in the direction of the resting point of accommodation (thedegree of accommodation of the lens when a person is in a dark room oris otherwise deprived of an adequate visual stimulus). This restingpoint is not at the ciliary muscle relaxation point, which would producea lens refractive power of 0 diopters, and varies between individuals.This inaccurate accommodation causes a video display to become somewhatblurred, and is thought to be a major contributor to the eye fatigue andheadaches that often accompany prolonged video display use. It wouldthus be desirable to reduce these inaccuracies in accommodation andthereby reduce a cause of eye strain.

Stereographic Video Displays: A number of video display manufacturershave attempted to increase the immersion and amount of information inthe display by creating stereographic displays (the term stereoscopicdisplay is often used interchangeably with this term). One example of astereographic display is the stereoscopic head mounted display (HMD).Typically, HMDs consist of a helmet or set of goggles, with a separatesmall LCD screen set in front of each eye. Lenses are mounted betweeneach LCD and eye and are typically configured to place the image atoptical infinity. The images displayed on each LCD are not identical,but instead represent slightly different camera viewpoints. The left LCDdisplays the left half of a stereo image pair and the right LCD displaysthe right half.

Another example of this display is the stereographic head trackeddisplay (HTD). Two versions of HTD are common. With some HTDs, a userdons lightweight LCD shutter glasses with lenses that become opaque ortransparent in synchrony with the frames displayed on a large table orwall mounted 2-D video display. When the shutter over the left eye isopened, the shutter over the right eye closes, and the left half of astereoscopic image pair is flashed on the display. When the shutter overthe left eye then closes, the shutter over the right eye opens, and theright half of the stereoscopic image pair is displayed. The opening ofshutters alternates from side-to-side in quick succession, and thedisplay synchronously shifts between showing left and right views, withevery other frame displayed on the monitor reaching each eye.

In other implementations of an HTD, the user wears glasses in which theleft lens is polarized along an axis orthogonal to that of the rightlens (e.g., the left lens is polarized vertically, while the right ispolarized horizontally). The user looks at a screen, upon which the leftstereo image has been projected with light polarized along one axis, andupon which the right image has been projected with light polarized alongthe other axis. Each lens admits only the light reflected from thescreen that is of the matching polarization, so each eye is presentedwith a different image. Other implementations of stereographic displaysinclude autostereoscopic displays, which enable users to view stereoimages without wearing any special glasses.

With all of these stereoscopic displays, the left eye receives only theleft stereo image, while the right receives only the right stereo image,giving the user the illusion that he/she is looking into a virtual scenewith two eyes at normal eye separation.

Current Stereographic Displays Violate Accommodation-VergenceSynkinesis: Current stereographic displays, especially HMDs, tend tocause profound eye fatigue, often accompanied by headache. Researchsuggests that this eye fatigue is, in part, the result of anincompatibility between the construction of the display and the biologyof accommodation. Specifically, the displays elicit a mismatch betweenthe depth information provided by accommodation and that provided byvergence. The displays have a fixed plane of focus (usually at opticalinfinity, or 0 diopters) making it necessary for the viewer's eyes tomaintain a static level of accommodation while using the display. If theviewer shifts his/her level of accommodation, the display becomes out offocus. Nonetheless, stereo displays also require that the viewerdynamically change the vergence of his/her eyes to binocularly fuseobjects at different apparent distances. Accommodation and vergence areyoked—that is, they are synkinetically linked, such that when a person'seyes converge, the eyes also tends to accommodate near, and when aperson's eyes deconverge, they tends to accommodate far. These displaysviolate this linkage and require a viewer to simultaneously convergehis/her eyes while maintaining a fixed level of accommodation.Accordingly, it would be desirable to provide a display that allowsaccommodation and vergence to shift synchronously, just as they do innatural viewing, and thus removes the primary source of stereo displayeye strain and loss of visual acuity.

SUMMARY OF THE INVENTION

The present invention employs a display method that, based on a measuredor estimated accommodation (focus) of the viewer, selectively appliesblur cues to images portrayed on a large DOF display. When using such adisplay in accord with the present invention, a viewer can look at adisplayed scene (real or virtual) and naturally shift focus betweenobjects in the scene. By accommodating to different objects, which maybe at different distances from a real or virtual viewpoint, thoseobjects come into sharp focus. The method gives the viewer the illusionof viewing a scene with objects at different depths in the scene, and indoing so, simulates natural viewing conditions. This method dramaticallyincreases the interactivity of the display, which enhances the perceivedrealism of the display. The ability to accommodate to objects portrayedon a display can increase the impression of realism and the feeling ofimmersion in a displayed scene, reduce eye strain (because the viewerdoes not maintain static accommodation), and reduce computation time,since a computing device coupled to drive the display needs to devotefewer resources to render an object that is out of focus. When appliedto stereographic displays, the system allows accommodation to move insynchrony with ocular vergence, reducing eye strain and the loss ofvisual acuity that results from a mismatch of accommodation and vergenceangle.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same becomesbetter understood by reference to the following detailed description,when taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a schematic diagram that illustrates how changes in the shapeof a crystalline lens do not affect the focus of VRD images, due to thelarge DOF created by narrow beams of light forming a narrow exit pupilin an eye;

FIG. 2 is a schematic view illustrating one method in accord with thepresent invention, for creating a large DOF display;

FIG. 3 is a schematic diagram showing one embodiment for configuring alarge DOF display and an accommodation monitoring device;

FIG. 4 is a schematic diagram showing another embodiment for configuringa large DOF display, with the accommodation monitoring device disposedperpendicular to the axis of the eye;

FIG. 5 is a schematic diagram showing yet another embodiment, with boththe large DOF display and accommodation monitoring device disposedperpendicular to the axis of the eye, allowing the viewer a line ofsight to the real world for augmented reality applications (virtual datasuperimposed over real world data);

FIG. 6 illustrates an embodiment in which a device to monitoraccommodation is disposed off-axis, and shows that an optional binocularmonitoring of vergence (relative to both eyes) can be made with asuitable device, to determine accommodation, while the viewer has a lineof sight to the real world for augmented reality applications;

FIG. 7 is another embodiment in which an off-axis monitoring ofaccommodation is made;

FIG. 8 is a flow chart showing the logic employed in the presentinvention to select and display an image with an apparent focus planethat corresponds to a viewer's level of accommodation;

FIG. 9 illustrates an array of 2-D images that are pre-prepared for usein the present invention; and

FIG. 10 is a schematic diagram illustrating an arrangement for real-timefocus control of a remote video camera with accommodation, wherein aviewer sees a real scene of a large DOF display, and accommodation ismonitored dynamically.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Construction:

The following components are needed to implement the present invention:(1) a display with a large DOF; (2) means for the (direct or indirect)measurement of accommodation; and, (3) a computing device that, at aminimum, receives the accommodation measurements, and can affect theinput to the large DOF display. Each of these components is discussed infurther detail below.

Large DOF Displays:

Any display with a large DOF can be used in this method. Severaldisplays that fit this description are summarized here, but this doesnot exclude other possible displays with a large DOF. The providedexamples of large DOF displays are not comprehensive, but rather serveas examples to illustrate the general principle of a large DOF display.Other large DOF displays can be used in place of the examples discussedbelow.

VRD: The VRD is one type of display that can be used as a large DOFdisplay in the present invention. As shown in FIG. 1, the VRD usedpreferably comprises a scanned light display 21 (also known as a retinalscanning display, a scanned laser display, a scanning light display, ascanning laser display, a retinal scanned light display, a retinal lightscanning display, a flying spot display). Scanned light display 21includes a modulated light source 22 and an X-Y scanner 24. Light fromthe scanned light display is directed toward a concave mirror 26 througha beam splitter 28. The beam splitter 28 directs light reflected fromconcave mirror 26 toward a viewer's eye 30. Because all of the raysprojected from the VRD converge at a very small exit pupil 36, theimages it projects have a large DOF. The display creates the equivalentof a “pinhole” aperture at its exit pupil, enabling it to project aclear image on the back of a viewer's retina 34 independent of thedegree of accommodation of a viewer's lens 32. In a view 40, a viewer'slens 32′ is flatter, showing an accommodation to view an image that isfarther away compared to lens 32 in a view 20, where lens 32 appearsmuch rounder, as would be the case in viewing an image that is closer.However, because the image viewed by eye 30 is provided with a largeDOF, the light rays that form the image are in focus on retina 34 inboth view 20 and view 40. Because all planes remain in similar focusthroughout the range of accommodation of the viewer's eye, a viewer isprovided with little or no feedback to the accommodative system, givingrise to open-loop accommodation.

Small Aperture Viewing: The DOF of any display can be increased byviewing that display through a small aperture (such as a “pinhole” in asheet of foil). Apertures of 0.5 mm diameter, or smaller, work best togenerate open-loop accommodation while viewing a display, but aperturesbetween 0.5 mm and 2 mm in diameter will partially open the loop,producing a satisfactory result. Viewing a display through a smallaperture reduces the total amount of light entering the eye, so it ispreferable to use fairly bright displays. If the device used to monitoraccommodation uses non-visible wavelengths of electromagnetic radiation,the small aperture can be in a filter that is opaque to visible light,but transmits infrared light or some other non-visible wavelengths ofelectromagnetic radiation.

Alternate Method for Creating a Large DOF Display: Very bright displays(including, but not limited to LCD projectors) can be modified to becomelarge DOF displays. As shown in FIG. 2, a pinhole aperture 52 can beplaced at some point in front of an image source 54 (e.g., an LCD).Depending on the variety of image source or video projector actuallyused, the optics of the projector may be in front of or behind the smallaperture, or may be removed completely. The beams emerging from thesmall aperture can be collected with optics 50 (which may comprise oneor more lenses) that redirect the beams to form a small exit pupil 36(an image of pinhole aperture 52). When the pupil of a viewer's eye 30is placed to coincide with exit pupil 36, the viewer sees a Maxwellianview of image 37 on retina 34 of the projected image, with a large DOF.

To adjust the divergence angle of the beams passing through the smallaperture, an additional set of optics 56 may optionally be placed infront of the aperture. FIG. 2 also shows the beam divergence (using thedotted lines on each side of one of the beams shown with dash lines). Bycomparing the beam characteristics in FIG. 2 with those of the VRD inFIG. 1, it can be seen that this display can be analogous to a parallelscan version of a scanned light display. That is, whereas the VRD scansa narrow beam serially, this alternative display projects all of thebeams in parallel. It should be understood that the method (andvariations thereof) described in this document for one of these displaysis also applicable to the other.

Means for Monitoring Accommodation:

Concurrent with the viewing of the large DOF display, the accommodationof the viewer must be monitored, to implement the present invention. Anumber of techniques can be used to monitor accommodation directly orindirectly. Some applicable devices include, but are not limited to, IRoptometers. These devices objectively determine the degree ofaccommodation by analyzing the reflection of the IR light that theyshine into a person's eye. Accommodation monitoring devices vary in thefrequency with which they can make measurements. The more frequentlythey can make measurements, the better suited they are for use in thepresent invention.

Direct Measurements of Accommodation: One example of an IR optometerthat can be used in this approach is the Welch Allyn, Model 14010SureSight™ autorefractor. However, any similar device that providesinformation about a viewer's level of accommodation can be used in theplace of the SureSight™ autorefractor. The SureSight™ autorefractorcontinuously measures the refractive power of the eye at 5 Hz. Themeasurement range for spherical refractive power is from +6.0 dioptersto −5.0 diopters, and an additional measurement of cylindricalrefractive power from 0 to +3.0 diopters along the axis of the cylinderis provided. One index of the level of accommodation of the viewer isthe spherical refractive power; however, the spherical equivalent(spherical refractive power added to half of the cylindrical refractivepower) generally provides a better index. The SureSight™ autorefractorhas a working distance of 14 inches.

The SureSight™ autorefractor is most accurate when its IR beam can enterinto the eye along the eye's visual axis. A number of alternateconfigurations allow this to occur, and a subset of these configurationsis illustrated in the embodiments of the present invention shown inFIGS. 3-7. In each of these Figures, a general layout of components isshown for a binocular display. These Figures are also helpful invisualizing the layouts for monocular displays by referring only to theright side of each Figure. One configuration that enables accommodationto be monitored while a viewer uses a large DOF display requires placingaccommodation monitoring device 68 (such as the SureSight™ autorefractoror an equivalent) directly in front of the viewer's eye to directlytransmit light from the eye through an optional lens 70. The image fromthe large DOF display is reflected into the eye with a beam splitter 66(or cold mirror), as shown in FIG. 3. The accommodation monitoringdevice takes accommodation measurements of an eye 62 (the viewers righteye in this example, but optionally a left eye 60 can be used), whilethe eye views the reflection of a large DOF display 54′ in theMaxwellian view. The light beams from display 54′ forms a small exitpupil or aperture that is aligned with the entrance pupil of the eye.Although two accommodation monitoring devices could optionally be used,it is not necessary to do so, because the accommodation levels of theright and left eyes are linked. Thus, for the viewer's left eye 60, onlya beam splitter 64 (or mirror) is used to reflect large DOF display 54into the left eye.

Alternatively, accommodation monitoring device 68 is placedperpendicular to the line of sight of the viewer, as shown in theembodiment of FIG. 4. A beam splitter or hot mirror 66′, which in thisembodiment preferably reflects only non-visible wavelengths of light butpermits visible wavelengths to pass unimpeded, is mounted at a 45 degreeangle, at the intersection between a viewer's line of sight and ameasurement axis of the accommodation monitoring device. Thisconfiguration enables the viewer to look directly through the beamsplitter and into the large DOF display, which is mounted along theviewer's line of sight, while the accommodation monitoring devicecontinuously measures the level of accommodation via the reflection fromthe beam splitter.

In another embodiment, which is shown in FIG. 5, both accommodationmonitoring device 68 and large DOF display 54′ are disposedperpendicular to the line of sight of the viewer's eye 62. Beam splitter66′ reflects the non-visible measurement signal from accommodationmonitoring device 68 into the eye and also reflects the measurementsignal from the eye back into lens 70 of the accommodation monitoringdevice. Beam splitter 66 (or a cold mirror), which transmits thenon-visible measurement beam, reflects light from large DOF display 54′into eye 62, enabling the viewer to superimpose the displayed imageswith real objects 75.

Other accommodation monitoring devices are accurate even when usedoff-axis. Accommodation monitoring device 69, which is of this type, canbe mounted at an angle from the line of sight of the eye and monitoraccommodation while the viewer views large DOF display 54′ via areflection in beam splitter 64 (or a mirror) or directly, as shownrespectively, in FIGS. 6 and 7. In FIG. 6, as in FIG. 5, the viewer canoptionally view real objects 75 through the display apparatus, whileviewing superimposed displayed images.

Indirect Estimates of Accommodation: Because vergence and accommodationare synkinetically linked, an alternative to measuring accommodationdirectly is to monitor the vergence angle of the eyes, and therebyindirectly estimate the accommodation of the viewer. Because changes inaccommodation and vergence are highly correlated, a direct measure ofvergence provides a satisfactory estimate of accommodation. Many devicesare available that can quickly track eye movements (and therebycalculate vergence angle), and such devices are a relatively cheapalternative to devices that measure accommodation directly. In any ofFIGS. 3-7, a vergence monitoring device can be substituted for theaccommodation monitoring devices 68 or 69.

Computing Device:

A computing device 132 (shown in FIG. 10) with custom softwarecomprising a plurality of machine instructions that are stored andloaded in memory for execution by a processor in the computing device,or a device with a custom hardware board (not shown) that implementsthese steps, receives measurements from the accommodation monitoringdevice and converts those measurements into measurements ofaccommodative power. The algorithm accesses a database, identifies theimage that best corresponds to a measured accommodative power, andoutputs that image to the large DOF display for immediate display to theviewer. This process is repeated each time the accommodation monitoringdevice collects a new accommodation measurement (e.g., five times persecond using the SureSight™ autorefractor for making accommodationmeasurements).

General Description of Method

The present invention uses simulated dioptric blur cues to “close”open-loop accommodation to a large DOF display. That is, a viewer usingour system can look at a displayed scene (real or virtual) and naturallyshift focus between the objects in that scene as if he or she wereactually there. This effect is accomplished by the following means:

The method employs a repeating cycle of steps, which are illustrated inFIG. 8. The cycle may begin with any of the steps, and within one fulliteration of the cycle, the blur information will be brought intoaccordance with the level of accommodation. Each iteration of the cycleupdates the blur information and thus, enables the blur content ofimages to dynamically change in accordance with the viewer'saccommodation.

While a person views the large DOF display, the accommodation level ofthe viewer is continuously monitored with an external instrument, suchinstrument as the IR autorefractor, which was discussed above. However,as noted above, any accommodation monitoring device that providesinformation about the level of accommodation can be used in the place ofthe autorefractor.

The accommodation measurements provide input for the computing devicethat selectively blurs objects in the scene, based on the current oranticipated level of the viewer's accommodation. For example, if theviewer accommodates to a distance of five meters, the computing deviceleaves objects near the five meters point in the scene unblurred andblurs other objects in proportion to their distance from the five metersfocus point. The computing device outputs the selectively blurred imageto the large DOF display.

The viewer sees an updated view of the scene with appropriate blurinformation. Although the displayed image remains sharply opticallyfocused on the viewer's retina at all times, the viewer perceivesportions of the image to be blurry as if they were optically defocused.As the viewer shifts accommodation, the display dynamically adjusts theblur cues (bringing some objects into focus, and others out of focus) tomatch the accommodation level.

The logical steps used in this process are shown in FIG. 8. In a step80, the person views the display and the person's eyes accommodate andconverge on an element in the large DOF display. The accommodation ofthe viewer's eye(s) is measured in a step 82. Alternatively, the systemmeasures the vergence (or gaze direction) of the viewer's eyes in a step82′ (and can then optionally compute the accommodation power based onthe vergence). In a step 84, the measurement made in step 82 or 82′ isadjusted by applying a predetermined calibration function, asappropriate. The calibrated result from step 84 is then used to quantifythe level of accommodation exhibited by the viewer's eye(s) in a step86.

A decision step 88 determines if the image focus plane provided by thelarge DOF display matches the viewer's accommodation. If so, the logicproceeds with a step 90 that determines the image provided by the largeDOF display need not be updated. Accordingly, in a step 92, the largeDOF display continues to show the image (without any change in the imagefocus plane). However, if the results in decision step 88 indicate thatthe image focus plane does not match the viewer's accommodation asmeasured, the blur information in the image that is displayed must bealtered.

There are a number of alternative means by which the computing devicecan control the blur information in the image. Three alternate paths (A,B, and C) are illustrated in FIG. 8, but other techniques canalternatively be used. Further details about these alternate methodsemployed by the computing device are discussed below. After one of thesethree approaches is used (i.e., after following the steps of path A, B,or C), the logic continues with a step 98 in which a new image isselected for display to the viewer, to provide the best focus planecorresponding to the current or anticipated level of accommodation bythe viewer. A step 100 then updates the image to be displayed with theselected image, and step 92 displays the updated image on the large DOFdisplay, as noted above. The logic then loops to step 80 to repeat theprocess.

Source Images:

A wide variety of images can be presented on the large DOF display. Thesource images of scenes can be obtained in a number of ways. A usefuldistinction can be drawn between two categories of images, i.e., theimages presented on the large DOF display can depict either virtualscenes, or real scenes. A second useful distinction can be drawn between2-D images derived from 3-D scenes in real-time, and pre-prepared 2-Dimages. The four combinations of these two categories lend themselves todifferent varieties of implementation. Each of these implementationvariants is discussed in greater detail below, and for eachimplementation, a path in FIG. 8 (A, B, or C) sketches its logicalimplementation.

Virtual Scenes—Real-time Rendering: In connection with the logical stepsof path A in FIG. 8, a virtual 3-D scene is created in or transferredinto a 3-D modeling program. Three-dimensional rendering software can beused to render 2-D images of the virtual scene and add variable amountsof blur to selected virtual objects (sometimes referred to as renderingwith “lens focus effects” or “DOF effects”). The 3-D rendering softwaresimulates the limited DOF of natural vision by adding blur to thosevirtual objects that lie outside of a chosen plane of best focus. Theplane of focus can be changed arbitrarily by a user or a program. In thepreferred implementation for virtual scenes, the scene is rendered inreal-time while the viewer uses the large DOF display and has his/heraccommodation dynamically measured. Every time a new accommodationmeasurement is taken, the plane of best focus in the 3-D renderer isdynamically determined and shifted to match the current accommodation ofthe viewer in a step 94 of FIG. 8, path A. In a step 96, the 3-Drendering software is then used to render a scene or image with a newfocus plane that matches that corresponding to the accommodationmeasured for the viewer. The logic then proceeds with step 98.

Using this implementation, the viewer can move and look around a virtualscene (using a joystick, mouse, motion trackers, or other input orpointing devices) and naturally accommodate to different objects in thevirtual scene. The location of the viewer in the scene, the gazedirection, and accommodation of the viewer all are used to dynamicallycontrol the rendering options for the scene.

While high resolution realistic real-time rendering requires significantcomputational power, use of accommodation information in the presentinvention can substantially decrease the computational power that isrequired for rendering the images displayed. The accommodationinformation can be used as a level of detail (LOD) effect, reducing thedemand on the processor. Other conventional LOD effects are based on thedistance from the person's viewpoint to other objects in a virtual scene(distant objects are rendered at a lower resolution than close objects).In these implementations, the accommodation of the eye provides anadditional cue for LOD effects. Only objects on the focal plane in thevirtual scene need to be rendered at the maximum resolution andcontrast. All other objects off the focal plane can be rendered at alower resolution and contrast, since the objects' high resolutioncontent is subsequently masked by blurring. The farther an object isfrom the focal plane, the lower the resolution and contrast that arerequired to render the object so that the object looks natural. Thismethod of using a viewer's accommodation information as a LOD effectreduces the total processing power necessary to render realistic highresolution 3-D scenes.

Stereographic images (one for each eye) can be generated by renderingthe virtual scene from two viewpoints. All of the methods described inthis provisional patent can be applied to stereographic images. Inaddition, for all of the approaches described herein for practicing thepresent invention, cylindrical or spherical image formats (such as theApple Corporation's Quicktime™ VR format), and 2½-D image formats can beused in place of 2-D images. Cylindrical or spherical image formatsrender a panoramic view of a scene and enable a viewer to rotate his/hervirtual view of a scene without requiring the system to render separateimages for each view rotation.

The present embodiment enables a viewer to dynamically rotate the viewof virtual scene without the scene needing to be re-rendered. Under thisembodiment, a new image only needs to be re-rendered when theaccommodation level of the viewer changes or the viewer moves within thescene. Two and one-half dimensional image formats retain the z-axis(depth) information for each pixel of a rendered image. Many 3-Drendering software packages separate the rendering of the 3-D scene fromthe rendering of the focus effects. First, they render a 2½-D image fromthe 3-D scene, then they render a selectively blurred 2-D image from the2½-D image. Using this method, the 3-D renderer only needs to render anew 2½-D image when the viewer moves within a scene or rotates the view.If the viewer changes accommodation, the 2½-D image remains the same,and only the focus effects applied to it are changed. In addition, ahybrid 2½-D cylindrical or spherical image format, in which acylindrical or spherical image contains z-axis (depth) information, canalso be used in place of 2-D images in the herein described methods. Inthis embodiment, the 3-D renderer need only render a new 2½-D image whenthe viewer moves through the scene or views moving objects. The focuseffect only needs to be re-applied to the 2½-D image when the vieweraccommodates—if the viewer rotates the view of the scene, the focuseffect need not be updated. These methods reduce the total computationnecessary to render a realistic high resolution 3-D scene.

Virtual Scenes—Pre-Rendering: A computationally efficientimplementation, which is suitable for use in connection with path B inFIG. 8, renders a series of images from the virtual scene offline, butallows real-time control of the playback of the pre-rendered 2-D images.Before the viewer uses the large DOF display, a virtual scene isrendered into a number of 2-D images. In the simplest implementation ofthis embodiment, the objects in the 3-D scene and the virtual cameraremain stationary across the series of rendered images. For each 2-Dimage, however, the location of the plane of best focus for therendering effect is different. From the static scene, a continuum ofimages is generated, with the focus at optical infinity on one end ofthe continuum, and with the focus at a near point (e.g. 1 cm) at theother end. A one-dimensional (1-D) array of 2-D images is created andsaved for future presentation. When the viewer uses the large DOFdisplay and accommodates, the computing device selects the image fromthe array that was pre-rendered with a focal plane that matches theaccommodation of the viewer, as indicated in a step 102 in FIG. 8, andsends it to the large DOF display for immediate presentation in a step104. The logic then proceeds with step 98, as described above. With thisembodiment, the viewer sees a still scene on the display and may freelylook around and shift focus between the various objects in the scene.

Another method to handle virtual scenes with pre-rendering is similar tothat discussed above, with the difference that a multi-dimensional arrayof 2-D images of a virtual scene is pre-rendered. As a simple example, a2-D array 110 is shown in FIG. 9 (higher dimensional arrays are alsouseful, but are more difficult to depict). Along one dimension of thearray, e.g., the horizontal axis, the position of the plane of bestfocus varies and in other dimensions, e.g., along the vertical axis,different aspects of the scene rendering vary. In the exemplary 2-Darray of FIG. 9, 2-D images are created so that along the firstdimension, the best focus plane varies and along the second dimension,the forward and backward position of the virtual camera varies. A thirddimension can be incorporated into the array to represent animation of(an) object(s) within the scene. Additional dimensions can be used torepresent more complex object animation, camera motion (e.g., pitch,yaw, translation, etc.), or other dynamic properties of the scene. Thisimplementation gives the viewer greater freedom to move around ananimated scene and dynamically accommodate to different objects, whilekeeping the demand on the processor of the computing device relativelylow.

For all of the embodiments described in this document, cylindrical orspherical image formats (such as the Apple Corporation's Quicktime™ VRformat) or 2½-D image formats can be used in place of 2-D images.Cylindrical or spherical image formats render a panoramic view of ascene, and enable a viewer to rotate his/her virtual view of a scene,without requiring the system to render separate images for each viewrotation. Two and one-half dimensional image formats retain the z-axis(depth) information for each pixel of a rendered image. In this case,the images are pre-rendered without focus effects and the focus effectsare rendered in real-time from the z-axis information. Many 3-Drendering software packages separate the rendering of the 3-D scene fromthe rendering of the focus effects. First, these software programsrender a 2½-D image from the 3-D scene, then they render a selectivelyblurred 2-D image from the 2½-D image. In addition, a hybrid 2½-Dcylindrical or spherical image format, in which a cylindrical orspherical image contains z-axis (depth) information, can also be used inplace of 2-D images in the present invention. Each of these image formatalternatives are compatible with the single and multi-dimensional arrayembodiments described in this document. As with the real-time renderingimplementation, stereographic images can be generated by pre-renderingthe virtual scene from two viewpoints. All of the embodiments describedherein can be applied to stereographic images. To conserve storagespace, the images can be compressed, and/or the arrays of images can beconverted into a video format and compressed using motion compressionalgorithms (such as that employed in the Moving Picture Experts Group(MPEG) algorithm).

Real Scenes—Pre-Captured Images: This embodiment is also relevant to thesteps of path B in FIG. 8. As shown in regard to a system 130 in FIG.10, one method of generating images of real scenes is to employ a camera136 (video or still, digital or analog) to acquire a number of imagesfrom a real 3-D scene 140, while the focus of camera lens 138 is variedacross the images, from nearest focus to optical infinity (so thatobjects on different planes of depth will come into focus and out offocus across the set of images). If an analog camera is used, thepictures are digitized and loaded into the storage of computing device132. The images are placed in an ordered array from farthest focus tonearest focus, and each image is coded to identify its best focusedfixation plane. A database is generated that links images in the arrayto estimated accommodation levels (e.g., an image taken with the camerafocused at infinity is linked with an accommodation estimate of 0.0diopters).

The operations of the computing device for this implementation aresimilar to those for the pre-rendered virtual scenes, supra. All of themethods previously described for acquiring, representing, and handlingimages of virtual scenes are also applicable to the pre-captured imagesof real scenes. For instance, the captured images can be formed intomulti-dimensional arrays to allow for greater interactivity with thedisplayed scene (e.g., the viewer and/or objects can move within ascene) and cylindrical and spherical image formats, 2½-D image formats,and 2½-D cylindrical and spherical image formats can be used in place ofstandard 2-D images.

Stereo images are generated by acquiring images from two viewpoints in ascene (two cameras can be used simultaneously, or one camera can bemoved between to viewpoints serially). The viewpoints can be equal inseparation to that of human eyes, or they can be farther apart toexaggerate the stereo information (or closer together to decrease thestereo disparity). In all of the methods mentioned herein, stereo imagescan be substituted for images.

Multiple cameras with different focal lengths can be used tosimultaneously acquire images from the same scene. A conventionalsoftware algorithm is used to interpolate intermediate images betweenthe acquired images from the different cameras (e.g., if one camera isfocused at 4 diopters and the other camera is focused at 5 diopters, thesoftware algorithm is used to interpolate images with a focus of 4.5diopters). This approach enables a set of images with varied focusplanes to be generated for a moving scene. When performed with a set ofstill cameras, the method creates a set of pictures of the samefreeze-frame of a moving scene. When performed with a set of videocameras, the method can create a set of movie channels that areidentical except for their focus planes. When the movies are playedback, the accommodation of the viewer controls the channel that isdisplayed on the large DOF display. Thus, the viewer is able to watch amovie, while dynamically accommodating to different objects in the moviescene.

Real Scenes—Real-time Camera Control: This embodiment is relevant to thesteps of path C in FIG. 8. One or more video cameras (digital or analog)are used to film real scene 140, generally as shown in FIG. 10. Thisfootage is relayed to large DOF display 54′ in real-time, so the viewercan see the scene from the camera's point of view. Video camera 136 canbe in the vicinity of the viewer, to show the viewer's own environment,or the video camera can be at a remote location, enabling telepresenceand teleoperation abilities. The focus of the camera is changed with anactuator 134. The computing device controls the focus by sending signalsto the actuator, and the focus is dynamically changed (a step 108 inFIG. 8) to match the accommodation level of the viewer, as indicated ina step 106 of FIG. 8. Two cameras can be used to create stereo videoimages. The accommodation of the viewer measured with accommodationmonitoring device 68 provides a signal to computing device 132, whichcontrols the focus for each of the cameras in response, as shown in FIG.10.

In addition to measuring accommodation, the motion of the viewer's headcan be tracked, and the orientation of the camera(s) (e.g., pan, tilt,yaw, etc.) can be actuated to follow the head motion. Optionally, thevergence angle of the eyes can control the vergence angle of thecameras. The zoom of the lens can be linked to an input device (e.g.,joystick or keyboard) to allow the viewer to move in and out of thescene. These features enable a viewer to look around a remote real scenein real-time, accommodate naturally to objects in the scene, and zoom inon objects.

Another embodiment uses a similar arrangement, with the exception thatthe accommodation of the viewer controls another mechanism of thecamera. For instance, the zoom level of the camera can be linked to theaccommodation of the viewer. When the viewer accommodates to a distantobject in the scene, the camera zooms in towards that object. In anotherembodiment of this concept, the accommodation of the viewer can be usedto control a linear actuator that physically translates a camera forwardand backward relative to the scene being recorded with the camera.

Other Modifications:

A number of exemplary implementations of the present invention have beendiscussed above. Combinations of any the elements from the exemplaryimplementations can be formed into many other implementation variants,which are too numerous to specifically describe. Other non-describedimplementations that combine elements of these described implementationsare also claimed.

All of the embodiments described above can be applied to stereographicimages, such as varying the focus of a stereoscopic pair of camerasinstead of only a single camera. In addition, for all of the embodimentsdescribed herein, cylindrical or spherical image formats (such as theQuicktime™ VR format) and 2½-D image formats can be used in place of 2-Dimages.

Integrated Large DOF Display with Infrared (IR) Eye Tracking: Theaccommodation and/or vergence monitoring device can be incorporated intothe large DOF display. They are listed as separate components in therest of this disclosure, but they can be combined into one physicalunit. This combination of components makes the system lighter, morecompact, and is expected to be more energy efficient.

For example using the VRD, constant intensity IR laser light can beadded to the modulated visible light source of the VRD before the beamsare scanned to the eye. The intensity and distribution of IR light beamsreflected from the eye will depend on the viewer's line of sight withinthe display. If the gaze of the eye is aligned with the collimated beamof IR light, then the angle of incidence of the incident beam will havezero degrees. After striking the apex of the viewer's cornea, thereflected beam will broaden while returning through the X-Y scanners tothe original source. Typically, the source is an optical fiber.Therefore, surrounding the source fiber can be one or more rings ofoptical fibers to collect the reflected beam of light. Only when the IRbeam of light is aligned with the viewer's line of sight will thereflected beam have a high intensity that is distributed symmetricallyabout the central axis, corresponding to the central source fiber. Thus,equal intensities of light should be collected in each optical fiberwithin each ring when the scanned beam of IR light is aligned with theviewer's line of sight. When the IR beam is not aligned with theviewer's line of sight, the reflected beam is not centered about thecentral source fiber among the ring of collection fibers. Accordingly,some collection fibers will have much lower intensities than collectionfibers on the opposite side of the ring. During each frame of thedisplay, the scanned IR beam and its distribution of reflected lightwill track which pixel the viewer is centrally fixated on. Bydetermining the scan position or pixel being fixated on by both left andright eyes of the viewer, the vergence angle can be calculated andaccommodation can be inferred.

A large DOF display with an array of multiple exit pupils can be used toenable users to change eye position within a large field of view displaywithout “losing” the single small exit pupil. To prevent multiple exitpupils from entering the viewer's eye, the array can be spaced apartsuch that only one exit pupil can enter the eye at any one time. Anotheroption is to turn off all exit pupils not centrally aligned with theviewer's gaze. Crude eye-tracking can be used to determine the exitpupil that should be active. Alternatively, eye-tracking can be used toactively keep a single exit pupil centrally aligned with the viewer'seye.

As people age, the crystalline lens becomes less elastic, and themaximum magnitude of their accommodative range decreases. This reducedability to accommodate or vary focus is termed presbyopia. The amplitudeof accommodation shifts of older viewers can be amplified, so that smallchanges in accommodation result in larger shifts of the simulated focalplane in the large DOF display. The total range of youthfulaccommodation can be mapped onto the restricted range of older viewers.For instance, if an older viewer can only accommodate to a maximum of 2diopters, that accommodation level can be linked with images displayinga 10 diopter focus level.

The accommodation of the viewer can be extended beyond the natural rangeof human eyes, such as looking and focusing very close to an object. Theability to maintain focus will produce a magnifying effect in a display,such as looking through a magnifying glass or microscope.

Non-natural fixation depths can be used to enable a viewer to look intoregions not normally accessible. For instance, a doctor can use a largeDOF see-through HMD (like that of FIG. 5) while examining a patient.Reconstructed data from imaging scans (e.g., fMRI, MRI, CAT, PET, EEG,MEG, EKG, x-ray, etc.) can be superimposed over the doctor's real viewof the patient. The doctor can accommodate to the level of the patient'sskin and see the real view of the skin. As the doctor shifts theaccommodation and vergence level in towards the inside of patient'sbody, an x-ray view of the patient's broken bone can be brought intoview. The interface can be configured such that the bone only comes intofocus when the doctor accommodates to its real depth. This provides thedoctor with an intuitive feel for the location of the bone break, andmay help the doctor find an internal site more rapidly and efficientlywhen performing surgery, for example, using laparoscopic techniques. Byapplying the natural oculomotor processes of accommodation and vergenceto non-natural data sets (like x-ray data), those data sets may beunderstood more easily. This approach can be used for general datavisualization applications. The accommodation and vergence levelprovides a natural means of controlling an additional variable whensorting through any data set.

Changes in Depth of Field: Using the methods described in this document,one can render images from a virtual scene with any desired depth offield. A DOF that approximates that of the normal human eye allows forthe most natural perception of a scene. A DOF that is more narrow thanthat of normal human vision may exaggerate the role of focus onperception of the scene. A hyperreal focus effect may be achieved. Therole of focus will be more salient, and this salience may help viewerswith accommodative dysfunction, such as the inability to focusaccurately, to improve their accuracy. By exaggerating the errors ofincorrect accommodation levels, the feedback to the accommodation systemis increased. This display system could be used as an accommodationtrainer for those who suffer from accommodative dysfunction.

Non-Display Applications: The focus level of the eye provides data tothe computing device about which objects are being attended to. Adisabled user could interact with real and virtual scenes by shiftingfocus to an object. A 3-D eye tracking system and 3-D mouse can beformed with the accommodation measurement method.

Simulated Longitudinal Chromatic Aberration: In addition to dioptricblur, the human accommodation system also receives feedback in the formof longitudinal chromatic aberration (LCA) when the eye accommodates.Short wavelengths (blue) are more strongly refracted than mediumwavelengths (green), which in turn are more strongly refracted than longwavelengths (red). Thus for a given level of accommodation, the red,green and blue channels of an image are not equally blurry. Thedifferences in blur across color channels provide the humanaccommodation system with additional feedback to control accommodation.All of the methods in this document that have been applied to blurinformation can be equally applied to LCA information. The LCA can besimulated for images in addition to (or as an alternative to) thesimulation of blur. Computing device images, like the colorphotoreceptors of the eye, are divided into red, green, and bluecomponents (or “channels”). In natural vision, the optics of the eye donot refract each wavelength of light to the same extent.

Although the present invention has been described in connection with thepreferred form of practicing it and modifications thereto, those ofordinary skill in the art will understand that many other modificationscan be made to the present invention within the scope of the claims thatfollow. Accordingly, it is not intended that the scope of the inventionin any way be limited by the above description, but instead bedetermined entirely by reference to the claims that follow.

1. A method for more accurately conveying depth in an image, comprisingthe steps of: (a) displaying an image to a viewer on a large depth offocus display wherein all elements in the image are initially displayedat an optical focus level that is substantially the same for allelements, thereby simulating an unnatural or artificial viewingcondition, since elements viewed at different distances from the viewerwould naturally appear at different focus levels; (b) determining anaccommodation for an eye of the viewer who is watching the image on thelarge depth of focus display, as a gaze of the viewer is directed towardan element in the image; and (c) displaying an image having an apparentfocus plane that tracks the accommodation of the viewer by addingblurring to other elements in the image in proportion to their distancein depth from the viewpoint of the viewer, so that as the accommodationof the viewer watching the large depth of focus display changes to focusthe eye of the viewer at a different viewing distance, the image that isdisplayed is changed to more accurately visually convey depth in theimage that is displayed, based on the accommodation that was determinedand thereby simulating a natural viewing condition.
 2. The method ofclaim 1, wherein the step of determining the accommodation comprises thestep of directly measuring the accommodation in at least one eye of theviewer.
 3. The method of claim 1, wherein the step of determining theaccommodation comprises the steps of: (a) measuring a vergence of atleast one eye of the viewer when watching the large depth of focusdisplay; and (b) determining the accommodation as a function of thevergence.
 4. The method of claim 1, wherein the step of determining theaccommodation comprises the steps of: (a) measuring a gaze direction ofthe viewer when watching the large depth of focus display; and (b)anticipating the accommodation of the viewer from the gaze direction. 5.The method of claim 1, further comprising the step of rendering inreal-time, each image having an apparent focus plane that tracks theaccommodation of the viewer, on the large depth of focus display.
 6. Themethod of claim 5, wherein objects within each image that are fartheraway from the apparent focus plane in the image are rendered at a lowerresolution and contrast, to substantially reduce a computationaloverhead required for rendering the image on the large depth of focusdisplay.
 7. The method of claim 1, further comprising the step ofpre-preparing a plurality of images having a range of different apparentfocus planes, so that the image having the apparent focus plane thattracks the accommodation of the viewer is selected from the plurality ofimages that were pre-prepared.
 8. The method of claim 7, wherein theplurality of images are arranged in a multi-dimensional array, at leastone axis of the multi-dimensional array corresponding to a dispositionof the apparent focus plane in the plurality of images.
 9. The method ofclaim 8, wherein each other dimension of the multi-dimensional arraycorresponds to a different parameter that varies within the plurality ofimages.
 10. The method of claim 9, further comprising the step ofenabling the viewer to provide an input that varies a value of aparameter for at least one of the other dimensions, to affect the imageprovided to the large depth of focus display.
 11. The method of claim10, wherein the parameter comprises one of: (a) a motion of a camerainto a scene comprising the plurality of images; (b) an orientation of acamera used to image a scene to produce the plurality of images; and (c)a zoom level of a camera used to produce the plurality of images. 12.The method of claim 8, wherein the image that is displayed by the largedepth of focus display is a 2½-dimensional image comprising visual depthinformation.
 13. The method of claim 7, wherein the plurality of imagesare pre-prepared by capturing a scene with a camera having a variablefocus set at a plurality of different focal planes.
 14. The method ofclaim 1, further comprising the step of producing the image having theapparent focus plane that tracks the accommodation of the viewer byadjusting a focus of a variable focus camera so that the variable focuscamera produces said image by imaging a real scene with the focus set atsaid apparent focus plane.
 15. The method of claim 1, further comprisingthe step of producing successive images having apparent focus planesthat track the accommodation of the viewer, at a sufficiently fast imagerate to produce a perception of motion of an object within thesuccessive images.
 16. The method of claim 1, further comprising thestep of producing an image having at least one element that is laterallyshifted and having an apparent focus plane that tracks the accommodationof the viewer, so that each eye sees a different image, to provide astereographic effect.
 17. The method of claim 1, further comprising thestep of employing a graphic rendering algorithm to blur objects that arenot disposed at the apparent focus plane in the image.
 18. The method ofclaim 1, wherein the step of determining the accommodation comprises thestep of employing light that is not visible to a human, to measure theaccommodation for the eye of the viewer.
 19. The method of claim 1,wherein the image that is displayed by the large depth of focus displayis in a non-planar format.
 20. A system for more accurately conveyingdepth in an image, comprising: (a) a large depth of focus display; (b)an image source that cooperates with the large depth of focus display toproduce an image that can be viewed; (c) a device that monitors at leastone eye of a viewer to produce a signal indicative of an accommodationof said at least one eye; and (d) a computing device coupled to theimage source and to the device, said computing device carrying out aplurality of functions, including: (i) displaying the image to a vieweron the large depth of focus display wherein all elements in the imageare initially displayed at an optical focus level that is substantiallythe same for all elements, thereby simulating an unnatural or artificialviewing condition, since elements viewed at different distances from theviewer would naturally appear at different focus levels; (ii)determining an accommodation for an eye of a viewer who is watching theimage on the large depth of focus display as a gaze of the viewer isdirected toward an element in the image; and (iii) displaying an imagehaving an apparent focus plane that tracks the accommodation of theviewer, so that as the accommodation of the viewer by adding blurring toother elements in the image in proportion to their distance in depthfrom the viewpoint of the viewer, so that as the accommodation of theviewer watching the large depth of focus display changes, to focus theeye of the viewer at a different viewing distance, the image that isdisplayed is changed to more accurately visually convey depth in theimage that is displayed, based on the accommodation that was determined,thereby simulating a natural viewing condition.
 21. The system of claim20, wherein the device emits light for directly measuring theaccommodation in at least one eye of the viewer.
 22. The system of claim20, wherein the device determines the accommodation by: (a) measuring avergence of at least one eye of the viewer; and (b) determining theaccommodation as a function of the vergence.
 23. The system of claim 20,wherein the device measures a gaze direction of the viewer, and thecomputing device anticipates the accommodation of the viewer based uponthe gaze direction.
 24. The system of claim 20, wherein in real-time,the computing device renders each image having an apparent focus planethat tracks the accommodation of the viewer, on the large depth of focusdisplay.
 25. The system of claim 24, wherein objects within each imagethat are farther away from the apparent focus plane in the image arerendered at a lower resolution and contrast by the computing device, tosubstantially reduce a computational overhead required for rendering theimage on the large depth of focus display.
 26. The system of claim 20,wherein a plurality of images having a range of different apparent focusplanes are pre-prepared, so that the image having the apparent focusplane that tracks the accommodation of the viewer is selected by thecomputing device from the plurality of images that were pre-prepared.27. The system of claim 26, wherein the plurality of images are arrangedin a multi-dimensional array, at least one axis of the multi-dimensionalarray corresponding to a disposition of the apparent focus plane in theplurality of images.
 28. The system of claim 27, wherein each otherdimension of the multi-dimensional array corresponds to a differentparameter that varies within the plurality of images.
 29. The system ofclaim 28, wherein the computing device responds to a user input thatvaries a value of a parameter for at least one of the other dimensions,causing a corresponding change in the image on the large depth of focusdisplay.
 30. The system of claim 29, wherein the image source comprisesa camera that is used to produce the plurality of images, and whereinthe parameter comprises one of: (a) a motion of the camera into a scenecomprising the plurality of images; (b) an orientation of the camerawhen imaging a scene to produce the plurality of images; and (c) a zoomlevel of the camera when producing the plurality of images.
 31. Thesystem of claim 27, wherein the image source displays a 2½-dimensionalimage on the large depth of focus display, so that a 2-dimensional imagecan be rendered by the computing device at a desired apparent focusplane using the depth information for the 2½-dimensional image, toreduce computational overhead.
 32. The system of claim 25, wherein theimage source comprises a camera having a variable focus, and wherein theplurality of images are pre-prepared by capturing a scene with thecamera with the variable focus set at a plurality of different focalplanes.
 33. The system of claim 20, further comprising an actuatorcoupled to a variable focus adjustment of a camera and to the computingdevice, said computing device producing the image having the apparentfocus plane that tracks the accommodation of the viewer by controllingthe actuator to adjust a focus of the camera so that the camera producessaid image by imaging a real scene with the focus set at said apparentfocus plane.
 34. The system of claim 20, wherein the computing deviceselects successive images having apparent focus planes that track theaccommodation of the viewer, at a sufficiently fast image rate toproduce a perception of motion of an object within the successive imagesviewed on the large depth of focus display.
 35. The system of claim 20,further comprising another image source that is coupled to the computingdevice and produces an image in which at least one element is laterallyshifted, said image having an apparent focus plane that tracks theaccommodation of the viewer, so that each eye sees a different image, toprovide a stereographic effect.
 36. The system of claim 20, wherein thecomputing device executes a graphic rendering algorithm to blur objectsthat are not disposed at the apparent focus plane in the image.
 37. Thesystem of claim 20, wherein the device uses light that is not visible toa human to measure the accommodation for the eye of the viewer.
 38. Thesystem of claim 20, wherein the image source displays an image on thelarge depth of focus display in a non-planar format.
 39. The system ofclaim 20, further comprising a beam splitter so that light from theimage source is reflected into an eye of the viewer, while light used bythe device for determining the accommodation travels between the deviceand the eye of the viewer through the beam splitter.
 40. The system ofclaim 20, further comprising a beam splitter so that light from theimage source is transmitted into an eye of the viewer, while light usedby the device for determining the accommodation is reflected into theeye of the viewer by the beam splitter.
 41. The system of claim 20,further comprising a beam splitter, said beam splitter reflecting lightfrom one of the image source and a real world scene, so that the viewercan simultaneously view the real world scene and the image provided bythe image source.
 42. The method of claim 12, further comprising thestep of rendering a 2-dimensional image at a desired apparent focusplane using the depth information for the 2½-dimensional image, if theaccommodation of the viewer has changed, thereby reducing acomputational overhead because the 2½-dimensional image does not have tobe re-rendered.
 43. The method of claim 12, further comprising the stepof re-rendering the 2½-dimensional image if either the accommodation ofthe viewer moves within a scene or the viewer views an object that ismoving in the scene.
 44. The method of claim 12, wherein the2½-dimensional image is either a cylindrical image or a spherical image.45. A method for more accurately conveying depth in an image, comprisingthe steps of: (a) displaying a 2½-dimensional image comprising visualdepth information to a viewer on a large depth of focus display; (b)determining an accommodation for an eye of the viewer who is watchingthe 2½-dimensional image on the large depth of focus display; (c)preparing a plurality of images arranged in a multidimensional array andhaving a range of different apparent focus planes, at least one axis ofthe multi-dimensional array corresponding to a disposition of theapparent focus plane in the plurality of images; (d) selecting the imagehaving the apparent focus plane that tracks the accommodation of theviewer from the plurality of images that were pre-prepared, said imagebeing a 2-dimensional image; and (e) as the accommodation of the viewerwatching the large depth of focus display changes, using the depthinformation of the 2½-dimensional image to display the 2-dimensionalimage, in order to more accurately visually convey depth, said use ofdepth information thereby reducing a computational overhead because the2½-dimensional image does not have to be re-rendered.
 46. A method formore accurately conveying depth in an image, comprising the steps of:(a) displaying a 2½-dimensional image comprising visual depthinformation to a viewer on a large depth of focus display; (b)determining an accommodation for an eye of the viewer who is watchingthe 2½-dimensional image on the large depth of focus display; (c)preparing a plurality of images arranged in a multidimensional array andhaving a range of different apparent focus planes, at least one axis ofthe multi-dimensional array corresponding to a disposition of theapparent focus plane in the plurality of images; (d) selecting the imagehaving the apparent focus plane that tracks the accommodation of theviewer from the plurality of images that were pre-prepared, said imagebeing a 2½-dimensional image; and (e) as the accommodation of the viewerwatching the large depth of focus display moves within a scene or theviewer views an object that is moving in the scene, displaying the2½-dimensional image having the apparent focus plane that tracks theaccommodation of the viewer, in order to more accurately visually conveydepth, based on the accommodation that was determined.